Many systems exist for enabling generation of summaries of a data steam, such as highlights of a program. These may be generated, for example, in TV broadcasting, to be shown to a user when the broadcast switches to a data steam not of interest to the user (such as a commercial block). The summary can be played back in place of the current broadcast. The summary may also be beneficial to the user if he or she misses a particular program or part of a program, in which case, the user can playback the summary of the program to catch up. It is a drawback of the known systems that their performance depends largely on the content being summarized. For example, certain systems are optimized for generating movie summaries and other systems are optimized for generating sports summaries.